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ABSTRACT 

An  algorithm  for  the  scheduling  of  information  acquisition  is 
explored  in  the  new  product  evaluation  context.   The  decision  process 
proposed  indicates  the  need  for  an  orderly  scheduling  process  in  con- 
junction with  a  specified  objective  function.   Simulations  employing 
a  variety  of  parameter  inputs  are  presented  for  illustrative  purposes. 


INTRODUCTION 

We  consider  the  decision-making  .ramework  to  involve  an  option  on 
the  decision-maker's  part  to  accept  an  alternative,  reject  an  alternative 
or  defer  the  decision  until  additional  information  is  collected  that  pre- 
sumably has  a  bearing  on  alternative  acceptance  or  rejection.   One  area 
dealing  with  information  collection  and  utilization  is  Bayesian  decision- 
making as  proposed  in  the  work  of  Raiffa  [6]  and  Schlaifer  [7],   (In  the 
marketing  research  context,  see  [l]).  Utilizing  this  framework,  work  in 
the  sequential  sampling  area  has  extended  the  "collect  information"  option 
to  the  situation  where  the  decision  maker  collects  information,  notes  the 
outcome  and  then  elects  to  accept,  reject,  or  collect  additional  informa- 
tion [3].   The  presumption  is  that  information  collection  is  a  costly 
operation  and  that  information  should  not  be  collected  beyond  the  point 
where  the  cost  exceeds  the  value. 

Previous  work  in  this  area  has  generally  involved  only  the  single 
parameter  decision  problem.   Consideration  of  a  decision  involving  many 
parameters  poses  an  additional  dimension  to  the  strategic  framework. 
Now  the  decision  maker  must  decide  between  accept,  reject,  and  informa- 
tion collection  in  which  he  has  a  number  of  alternative  parameters  to 
estimate.   If  we  use  the  word  "experiment"  to  refer  to  information 
collection  for  the  sake  of  parameter  estimation,  then  it  becomes  crit- 
ical to  specify  the  first  experiment  to  be  performed  as  well  as  a  sub- 
sequent experimental  order  which  is  dependent  on  the  prior  experimental 
outcomes. 

Since  only  the  experimental  outcomes,  and  not  the  order  of  com- 
pleted experiments  is  critical  to  subsequent  experimental  selection, 


a  dynamic  programming  recursion  algorithm  will  be  proposed.   Basic  inputs 
are  the  same  as  in  any  single  parameter  Bayesian  decision  framework:   the 
cost  of  each  experiment,  the  prior  pxobabilities  of  the  unknown  parameter, 
conditional  probabilities  that  relate  experimental  outcomes  to  the  true 
value  of  the  parameter,  and  an  objective  function  specifying  relationships 
between  variables  and  the  objective. 

The  problem  of  multi-parameter  estimation  are  neither  trivial  nor 
infrequent  in  management.   An  aircraft  manufacturer  has  a  number  of  tests 
that  may  need  to  be  performed  on  a  new  or  modified  product;  the  experi- 
ments are  costly  and,  thus,  savings  with  this  proposed  solution  may  be 
realized.  New  product  evaluations  are  also  most  difficult,  and  the 
number  of  experiments  range  from  inexpensive  breakeven  analysis  to  ex- 
pensive market-testing.   The  high  rate  of  failure  among  new  products 
indicates  that  some  products  should  have  undergone  more  extensive  ex- 
perimentation prior  to  experimentation.   The  large  number  of  products 
that  are  abandoned  after  market  testing  causes  one  to  question  whether 
these  products  could  have  been  screened  out  prior  to  expensive  experi- 
mentation.  Finally,  one  must  wonder  about  the  possibly  larger  number 
of  products  that  would  have  been  successful  but  were  abandoned  when 
firms  faced  the  high  cost  of  further  testing;  this  statistic  will  never 
be  known. 

Solutions  to  the  problem  of  new  product  evaluation  has  been  pro- 
posed [2,5],   Unfortunately,  these  provide  only  a  "rule  of  thumb"  for 
the  scheduling  of  the  steps  (see  Figure  1) ,  or  methodology  for  perform- 
ing each  experiment.   In  general,  the  steps  scheduled  earliest  are 
those  which  are  least  expensive;  one  might  also  expect  the  earlier 
steps  to  be  very  efficient  in  eliminating  poor  ideas.   Nevertheless, 
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the  assumption  of  this  prior  work  is  that  all  steps  will  be  followed  until 
rejection  and  that  the  order  specified  will  be  rigidly  adhered  to. 

An  Example 

To  continue  the  new  product  evaluation  example ,  assume  that  a  new 

product  development  team  formulated  the  following  objective  function  for 

its  new  product  idea: 

T 
(1)  Z  -  il -A— _  pS  [D   (50  -  C  )  -  FC  ]}  -  K 

t-0   (1  +  i)  u 

where, 

PS  «  probability  of  technological  success  (i.e.,  prob- 
ability of  developing  a  manufacturable  prototype) 

D  =  demand  for  new  product  (if  developed)  in  year  t 

C  =»  unit  variable  cost  associated  with  the  production 
of  the  new  product  in  year  t 

FC  =  fixed  costs  associated  with  new  product  in  year  t 
i  ■  cost  of  capital 

K  =  a  minimum  return  or  cost  of  implementing  the  new 
product 

Thus  we  see  that  the  new  product  team  faces  uncertainty  about  the  new 
product  idea  which,  if  developed,  would  sell  for  $50  per  unit.   It  is 
also  apparent  that  the  timing  of  cash  flows  is  critical  (this  is  util- 
ized in  present  value  calculations)  and  that  some  expenditures  may  vary 
over  the  years.   For  example,  fixed  cost  expenditures  are  typically  very 
heavy  in  year  zero  (e.g.,  purchase  of  capital  equipment)  while  demand 
and  manufacturing  costs  are  incurred  over  the  life  of  the  product. 


The  objective  function  featured  in  the  example  is  indicative  of  why 
analytical  solutions  to  problems  of  this  nature  may  not  be  feasible.   Ex- 
pansion of  equation  (1)  yields  a  term  which  is  the  interaction  of  three 
parameters,  PS«D  «C  .   The  value  of  reducing  uncertainty  for  C  ,  for 
example,  is  dependent  upon  the  parameter  estimates  and  uncertainty  about 
PS  and  D  .   Although  variables  may  clearly  interact  in  the  objective  func- 
tion, the  assumption  is  made  that  experimental  outcomes  for  different 
parameters  are  independent.   This  assumption  is  made  to  simplify  the  sim- 
ulations presented  in  a  later  section;  it  is  by  no  means  limited  to  the 
general  method  of  solution  and  the  program  could  be  easily  modified  to 
handle  the  condition  of  dependence. 

The  Programming  Approach 

To  expand  on  the  proposed  approach  to  the  multiparameter  sequential 
information  collection  problem,  consider  a  decision  situation  with  four 
unknown  parameters,  X. ,  X-,  X„,  and  X,  .   Let  us  introduce  the  following 
notation  and  decision  Inputs:  If  the  alternative  is  accepted,  our 
profit  will  be 

P  (X„ ,  X„  .  .  .  X  ) 

where  X. ,  X_,  X_,  X,  .  .  .  are  the  actual  (unknown)  values 
of  the  parameters 
We  assume  that  the  following  data  can  be  determined: 

(a)  c.,  the  cost  of  performing  an  experiment  on  the  i 
parameter 

(b)  fjfajJ,  the  prior  density  function  of  the  i 
parameter 


(c)  h, j   (x.),  the  conditional  density  function  of  the 
outcome  of  the  experiment  on  parameter  i,  x, ,  given 
the  true  value  of  the  parameter  is  x , . 

Then  (b)  and  (c)  above  can  be  used  with  Bayes  theorem  to  compute, 
g .  i *    (x,),  the  conditional  density  function  on  parameter  i  given  the  ex- 
perimental outcome  x , . 

Next  we  develop  the  dynamic  programming  recursion  to  determine  the 
optimal  order  [4].   First  consider  the  situation  when  our  objective  func- 
tion has  four  parameters  and  all  experiments  have  been  completed  with 

A  A  A  A 

experimental  outcomes  x. ,  x?,  x„,  x,.   Since  we  must  now  reach  a  terminal 
decision,  we  would  calculate  expected  profit  with  acceptance  as: 

(2)  E  (P|xlfx2,x3,x4)  - 

P(xnx0x0x,  )  g,  i-  (x, )  .  .  .  g,  i  ~   (x, )  dx,  .  .  .  dx. 

•Ix.^xJx.Jx,    12  3  4  °1  x,  v  1      a4  x.   4    1      4 
12  3  4  '1  '4 

The  value  associated  with  rejection  is  0.   Thus  the  decision  with  the 
greater  expected  value  would  be  selected. 

Working  back  one  step,  consider  the  case  where  only  three  experi- 
ments, 1,  3,  and  4,  have  been  completed  with  experimental  outcomes 
x  ,  x~,  x,;  three  options  are  available:  we  can  accept,  reject  (with 
expected  value  of  0)  or  perform  experiment  2.   The  expected  value  of 
immediately  accepting  is, 

(3)  A  -  E  (P|xr  x3,  x4)  - 

Jx1Ix2ix3ix4  P<xix2X3V  SlIx^V  f2  (x2}   «3|x3(ac3)   84|x4(x4>  dxl  "  '  -dx4 

or  utilizing  equation   (2)  we  can  rewrite  equation   (3)   as, 

(4)  A  -  J\     E   (F|xlf52,i3t54)    [J     h2|      (x2)   f2    (x2)  dx2]   dx2 


The  above  is  merely  performing  the  integration  over  a  different  variable 
and  the  quantity  in  brackets  represents  the  pre-posterior  distribution 
of  x_.   In  addition  to  the  terminal  act  options,  we  may  elect  to  perform 
experiment  2,  thereby  delaying  our  terminal  act,  but  also  incurring  the 
cost  of  the  second  experiment,  c? : 

(5)  C  -  -c2  +  Jj  max  [E(P  jx^.x^) ,  0]  [Jx  h2|x  (x2)f2(x2)dx2]  dx2 

It  is  apparent  that  the  cost  of  c«  must  be  weighed  against  the  advantage 
of  choosing  the  better  terminal  decision  after  observing  x_. 

To  simplify  future  expansion,  let  us  introduce  the  following 
notation: 

Let  K  represent  the  set  of  indices  of  all  possible  experiments 
K  -  {1,  .  .  .  ,  n} 

Let  J      represent  the  set  of  experiments  which  have  been  performed 

x )  k  j  in 

J .  .  m  =  K  -  {i,k,m}    (i.e.,  all  but  i,  k,  m  have  been  performed) 

i.e. ,  for  n  *  4 

K  -  {1,2,3,4} 

J2   ,    s  K  -  {2S4}  ■  {1,3}        (i.e.,   experiments  2  and  4  have 
'  not  been  performed) 

Let  Y„  -  {x.  .  .  .  x   } 
Kin 

And  Y_         «■   {x.  |  ie  J?   ,}  -   (x.  ,x~}      (i.e.,    the  experimental  outcomes 
2,4  for  the  experiments  performed) 

Let  Ay  ■  the  expected  value  associated  with  termination  by  acceptance 
given  experimental  outcomes  Y 
Then 

\         *  JxixJx  ixA  P(X1X2X3X4>   *1 1 *.<*!>    f2    (X2> 
J2,4  12     3     4  '1 

g3|x3(x3)   f4(V   <^i^2dx3dx4 


For  ieJ  let 

C     ■  the  expected  value  associated  with  continuing  with 
i,Yj 

experiment  i. 
Thus, 

C4,YT    "  "C4  +  &■»  (AY.   »  R>  €29YT     } 
J2S4  J2  2 


[   h.  |   (x.)  f/Cx,)  dx.]  dx, 
Jx,  4  x.   4   4%  4    4J   4 

4   '  4 

where 
R  =»  0 

Intuitively  the  above  integral  considers  the  expected  value  of  the  best 
decision  associated  with  each  experimental  outcome  x, ,  and  weights  the 
expected  value  by  the  probability  of  the  x.  outcome.  These  are  then 
summed  over  all  possible  xy  outcomes  to  yield  the  expected  value  of 
continuing  with  the  fourth  experiment. 

The  general  form  of  the  dynamic  programming  recursion  can  now  be 
written  for  the  n  parameter  case: 

1.  start  with  J  =  K~{1,  ...n>  and 

compute  A,   for  all  possible  Y 

\J  J 

Compute  EV   »  max  {^  ,R} 
J         J 

2,  Let  J  =  {K}  -  {i}.  Then  for  all  i 

compute  A,^   ,  C     ,  and  let 
YJ.    1?1J, 

EVY   -  max  {Ay  ,  R,  C±   Y  } 

i  x         i 

This  represents  the  case  where  only  one  experiment  remains  to 

be  done, 


3.  Let  J,  .  -  J,  -  {k}  »  J,  -  {i}.   For  all  k  *   i 

i,k    i         k 

we  then  compute  A^    ,  C 

J       *  J 
Ji,k      Ji,k 

Then  EV      -  max  {A     ,  R,  C       } 
T  I  x'  T 

Ji,k         Ji,k        Ji,k 

4.  Continue  to  recursion  to  and  including  the  case  J1       .   This 
represents  the  case  when  none  of  the  experiments  has  been  per- 
formed and  the  decision  maker  has  selected  the  best  option: 
either  accept,  reject,  or  continue  (and  the  best  experiment  to 
perform  first) . 

It  should  be  noted  that,  in  the  foregoing,  we  have  assumed  all  density 
functions  are  for  continuous  random,  variables.  Clearly,  if  this  is  not  the 
case,  then  we  merely  replace  the  appropriate  integral  with  the  corresponding 
summation.   From  a  computational  point  of  view,  however,  we  must  assume 
that  we  have  discrete  density  functions,  at  least  for  all  but  the  most 
trivial  situations.   Thus  to  perform  the  actual  computations  outlined 
above,  we  would  approximate  all  density  functions  by  discrete  density  func- 
tions . 

Sample  Runs  of  Experimental  Scheduling 

To  explore  the  impact  and  provide  a  qualitative  feel  for  the  model's 
scheduling  algorithm,  eight  runs  of  the  model  are  presented  in  Table  1. 
These  runs  embody  a  number  of  input  parameters  from  which  generalizations 
will  be  discussed.   To  conform  to  the  previous  example  as  well  as  the  program- 
ming notation,  we  considered  a  four  experiment  case  in  which  the  decision 
maker  must  determine  whether  to  obtain  information  regarding  the  probability 
of  technological  success,  demand,  unit  cost,  or  fixed  cost  for  a  potential 
new  product.   Thus  we  have  the  following  objective  function: 


10 


10       1 

2  =  E   i__^  PS  [D   (50-C  )  -  FC  ]  -  5000 

t-0  (l+i)C       t 


where  all  variables  are  defined  as  before  and 
i  -  .10 
PS  ■  a  discretely  distributed  random  variable 
D   C  =  normally  distributed  random  variables, 
equal  to  0  in  period  0  and  constant 
over  time  periods  1  to  10. 
FC  =  uniformly  distributed  random  variable  in 

period  0  and  equal  to  0  in  periods  1  to  10. 

The  computations  were  performed  by  dividing  the  range  of  the  density 
function  into  intervals  and  then  massing  the  total  probability  in  each 
interval  at  the  midpoint  of  the  interval.   The  conditional  distributions 
h.i   (x. )  were  approximated  by  massing  all  the  probability  in  an  interval 

i|X^   1 

at  the  midpoint.   In  particular,  the  range  for  D  was  divided  into  deciles 
and  the  ranges  for  FC  and  C  were  divided  into  quint iles.   Obviously,  as 
we  use  finer  intervals,  we  will  get  a  better  approximation,  but  our  cost 
of  computation  will  also  rise.   In  practice,  we  would  have  to  check  that 
the  discrete  distributions  reasonably  reflect  our  initial  data.   No  trans- 
formations were  necessary  in  the  case  of  PS  as  this  variable  was  initially 
a  discrete  variable. 

Table  1  provides  a  summary  of  the  input  and  output  of  the  computer 
runs.   For  experiment  1,  the  cost  of  the  experiment  and  prior  probability 
of  success  is  indicated  (the  probability  of  failure  is  simply  1  minus  this 
value)  as  well  as  the  probability  of  ultimate  success  given  a  successful 
outcome  of  the  experiment  and  the  probability  of  ultimate  failure  given 
an  unsuccessful  outcome  of  the  experiment.   The  latter  two  values  capture 
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the  reliability  of  experiment  1  in  predicting  ultimate  technological 
success.   For  the  remaining  three  experiments,  input  parameters  are  the 
cost  of  the  experiment,  the  prior  mean,  the  prior  standard  deviation, 
and  the  posterior  standard  deviation.   For  the  sake  of  simplicity,  it 
was  assumed  that  the  posterior  distribution  was  from  the  same  family  as 
the  prior  distribution  and  that  the  experiment  was  unbiased  (e.g., 
E  (XJx)  =*  x,  where  X  is  the  basic  parameter  and  x  is  the  experimental 
outcome) . 

Run  1  represents  the  base  run  from  which  generalizations  regarding 
the  other  simulations  will  emerge.  As  indicated  in  the  summary  table, 
experiment  1  is  the  first  experiment  scheduled;  depending  on  the  experi- 
mental outcome,  experiments  2  and  4  have  probabilities  of  being  scheduled 
second  of  .525  and  .475  respectively.   Figure  2  is  an  abbreviated  version 
of  the  entire  decision-experimentation  tree  for  the  base  run.   This  de- 
cision tree  is  indicative  of  the  infeasibility  of  analytical  solutions 
or  anything  but  broad  generalizations  to  this  method  of  problem  solving. 
For  example,  experiment  1  is  scheduled  first  because  of  the  reliability 
of  the  experiment  in  predicting  success  or  failure  as  well  as  the  com- 
paratively low  cost.   If  a  failure  is  predicted,  then  we  continue  with 
experiment  4  which  leads  to  terminal  acts  of  either  acceptance  or  re- 
jection.  If  the  outcome  of  experiment  1  points  to  success,  then  we 
continue  with  experiment  2.   If  demand  is  estimated  to  be  very  low 
(first  or  second  decile)  or  very  high  (tenth  decile),  we  terminate. 
The  decision  tree  also  provides  a  visual  presentation  of  the  non- 
monotonic relationship  between  outcomes  of  experiment  2  and  the  optimal 
third  experiment.   If  the  experiment  2  outcome  is  In  the  3rd,  4th,  8th, 
or  9th  deciles,  we  perform  experiment  4;  outcomes  in  the  5th,  6th,  or 


Figure  2 
Partial  Decision  Tree  of  Base  Run  (Run  1) 
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7th  deciles  warrant  experiment  3.   Thus  analytical  solutions  or  guesswork 
are  likely  to  yield  non-optimal  decision  procedures.   A  comparison  of  ex- 
pected values  of  the  alternative  decision  rules  indicates  that,  for  this 
set  of  parameters,  an  approximately  10  per  cent  gain  can  be  expected  from 
using  the  algorithm  as  compared  with  running  all  tests  and  then  choosing 
the  best  course  of  action.   The  algorithm  results  in  an  even  more  sub- 
stantial gain  when  compared  with  the  simple  decision  rule  to  do  no  ex- 
perimentation. 

A  comparison  of  runs  1  and  2  Indicates  the  eftect  of  experimental 
cost.  As  the  cost  of  experiment  2  is  increased,  the  scheduling  moves 
the  experiment  from  a  likely  second  step  to  a  possible  third  step. 

In  runs  3  and  4,  the  prior  standard  deviation  of  the  parameter  2 
value  has  been  decreased  from  1667  to  1250  and  625  respectively.   In 
this  fashion,  as  the  prior  standard  deviation  approaches  the  posterior 
standard  deviation,  the  experiment  has  either  a  low  probability  of  taking 
place  or,  in  the  case  of  run  4,  no  chance  at  all  of  being  completed.   Sim- 
ilarly when  experimental  reliability  is  decreased  (I.e.,  posterior  standard 
deviation  is  increased)  as  in  run  5,  the  probability  of  the  experiment 
being  done  diminishes. 

The  previous  runs  have  shown  how  experimental  costs  and  value  of 
information  have  moved  experiment  2  from  a  likely  second  step  to  an  im- 
probable third  step.   In  a  similar  manner,  experiment  1  (which  was  sched- 
uled first  in  the  base  run)  can  be  shifted  out  of  the  first  scheduling 
position.   In  run  6,  the  cost  of  experiment  1  has  been  increased  so  that 
experiment  1  Is  a  possible  second  step.   Run  7  illustrates  the  effect  of 
the  prior  parameter  of  experiment  1;  in  this  case,  the  likelihood  of  a 
success  has  been  increased  from  .5  to  .75  and  experiment  1  has  only  a 
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a  very  small  probability  of  occurring  before  the  fourth  step.   Run  8  in- 
dicates the  effect  of  decreased  information  reliability.   In  this  case, 
experiment  1  has  no  chance  of  being  within  the  first  three  experimental 
steps. 

Conclusion 

The  dynamic  programming  Bayesian  approach  is  a  flexible  technique 
that  may  be  applied  to  a  number  of  decision-information  acquisition  prob- 
lems that  involve  multiple  testing  of  several  parameters.   One  obvious 
use  is  in  the  area  of  new  product  evaluations  which  must  consider  a  number 
of  relevant  parameters  to  the  overall  profit  function.   Extensions  to  the 
areas  of  research  and  development  or  medical  testing  may  yield  additional 
promise.   The  nature  of  the  objective  function  is  quite  flexible  and  the 
independence  of  tests /parameter  estimation  is  by  no  means  limiting. 

The  value  of  the  organized  problem  solution  is  threefold: 

1.  Management /researchers  must  translate  qualitative  judg- 
ments about  the  quality  of  information,  prior  information, 
and  cost  of  information  into  quantitative  parameter  inputs. 
Where  high  uncertainty  exists,  sensitivity  analysis  is 
available  to  detect  parameters  that  greatly  affect  the 
scheduling  outcome. 

2.  The  nature  of  possible  complex  objective  functions  suggest 
that  analytical  or  rules  of  thumb  solutions  to  this  prob- 
lem will  result  in  less  than  optimal  decision  rules.  (See 
[8]  for  a  description  of  subjects'  inabilities  to  make 
judgments  of  this  nature  for  a  simple  problem.) 
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3.   The  process  only  schedules  information  that  will  have  a 
bearing  on  the  final  decision  and  the  ambiguity  or  poten- 
tial source  of  conflict  in  test  evaluation  is  thereby 

eliminated.    .  \ 
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